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ABSTRACT 

This study examined the concerns and confidence of 10 
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from class discussions. Group results from the SCQ indicated a shift 
from *’self” to ** * * impact** concerns. Confidence ratings increased over 
time; narratives showed that participants were confident in their 
abilities to implement naturalistic assessment but were not confident 
about the success of naturalistic assessment beyond their classrooms. 
Factors that influenced confidence were identified and categorized 
as: (1) teacher uniqueness; (2) professional development; (3) 
significant others; and (4) teaching environment. The participants 
felt that naturalistic assessment empowered them as well as their 
students. Portraits were developed for each participant that 
demonstrated the importance of a supportive teaching environment to 
the success of naturalistic assessment. (Contains 1 table, 3 figures, 
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NaturjJistic Assessment: Teachers’ Concerns and Confidence 

Abstract 

This study examined the concerns and confidence of ten educators as they adopted 
naturalistic assessment procedures. Participants enrolled in a four month graduate 
course implemented various literacy assessment procedures in their classrooms. Data 
were collected through The Stages of Concern Questionnaire (SoCQ), interviews, 
confidence rating and written narratives, and fieidnotes from class discussions. Both 
qualitative and quantitative methods were used for data analysis. Group results from the 
SoCQ indicated a shift from "self" to "impact" concerns. Confidence ratings increased 
over time; narratives showed that the participants were confident in their abilities to 
implenient naturalistic assessment but were not confident about the success of 
naturalistic assessnrfent beyond their classrooms. Factors which influenced confidence 
were identified and categorized: teacher uniqueness; professional development; 
significant others; and teaching environment. The participants felt that naturalistic 
assessment empowered them as well as their students. Portraits were developed for each 
participant which denwnstrate the importance of a supportive teaching environment to 
the success of naturalistic assessment. 
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Naturalistic Assessment: Teachers' Concern and Confidence 

As naturalistic assessment methods (Moore, 1 983) are incorporated into 
classrooms, teachers are changing their educational practices. Depending on the locale, 
naturalistic assessment procedures, such as literacy portfolios, running reading 
records, kidwatching, etc., may be required by administrators in the same way that 
standardized tests have been in the past. For some teachers, the change may be too 
abrupt especially if appropriate inservice education has not occurred. In other 
instances, classroom teachers have taken the initiative to change assessment practices; 
they demand that naturalistic methods are accepted by administrators and boards of 
education as valid ways of knowing about student success. In either case, the issues of 
teachers' concerns about new assessment methods and their confidence in using them in 
their classrooms are brought to the forefront as attempts to change current educational 
practices are made. As with other educational innovations, the success of its 
implementation rests to a large degree with teachers' acceptance of naturalistic 
assessment as well as an educational environment which allows change to occur (Olson, 
1985). 

As a teacher educator, I am interested in the change process when teachers 
encounter an educational innovation. Teacher educators piay an important role in the 
success of an innovation by helping teachers and administrators understand the process 
of change when they try to incorporate an educational innovation into the existing 
system. The purpose of this study, therefore, was to understand the process of change 
from primarily from individual teachers' perspectives, specifically when naturalistic 
assessment procedures were involved. 

Rather than the term alternative assessment which is popular in the literature 
today, I have chosen to use the term naturalistic assessment. Its use implies that 
assessment takes place naturally in a classroom environment and that students are 
evaluated through naturally occurring tasks and activities. Moore (1983) originally 
defined naturalistic assessment in relationship to reading comprehension: 

Naturalistic assessment of reading comprehension is based on 

observing students' responses to reading situations during the 
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school day. Educators through the years have given this approach 
labels such as ongoing evaluation (Austin, 1 958), diagnostic 
teaching (Strang, 1 964), kidwatching (Goodman, 1 978) and 
diagnosis by observation (Cunningham,! 982). In this approach, 
teachers observe students' behaviors in a variety of circumstances 
and with various materials, (p. 965) 

In my graduate class the definition was applied to all aspect of English/Language 
Arts; portfolios and performance assessment were considered naturalistic assessment 
procedures. 

Theoretical Framework 

Two theoretical perspectives formed the foundation for this study. The first 
perspective was the "new science of educational assessment" (Beilak, Newmann, Adams, 
Archbald, Burgess, Raven, & Romberg, 1 992). This contextual paradigm of assessment 
recognizes: the existence of multiple meanings of text within a multicultural society; the 
use of alternative methods for achieving construct validity; the inseparability of 
cognition, affective, and conative (persistence, determination and will) learning; and the 
need for local involvement and control of assessment. From this perspective, the 
current parametric paradigm, responsible for the institutionalization of standardized 
and criterion-referenced tests, must be challenged because it is a "major barrier to the 
renewal and restructuring of the nation's schools" (Beriak, 1992, p.l2). 

The second theoretical perspective which influenced this study was grounded in 
the research on teacher change and educational innovations (Hord, Rutherford, Huling- 
Austin, & Hall, 1987; Smylie, 1988; Fullan, 1990, 1991; Ohihausen, Meyerson & 
Sexton,! 992; Meyerson, 1993; Sarason, 1993). The basic concepts which influenced 
this study are: (1) change is idiosyncratic, complex, and ongoing and is greatly 
influenced by teachers' level of confidence or self-efficacy; (2) teachers are more 
willing to embrace innovations when they view new practices as better ways to meet the 
needs of students; (3) practices and teacher beliefs are interrelated; and (4) 
innovations are successful when teachers are supported in the change process over a long 
period of time. It is from these two perspective that the study was designed and 
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interpreted. 

Methodology 

During Spring, 1 994, ten experienced educators (K-1 2) were enrolled in my 
graduate course, "Naturalistic Assessment in English/Language Arts”. The participants 
were empbyed by the same large school district located in the Southwest and taught in a 
variety of situations at the elementary, middle, or high school levels. Nine of the 
participants were teachers; one was an elementary school principal who wanted to 
initiate naturalistic assessment methods in his school. All the educators were familiar 
with holistic approaches to literacy and had taken a course either in whole language or 
writing process previously. Several teachers were already implementing portfolios in 
their classrooms when the course began. Two high school English teachers used holistic 
and trait scoring to evaluate student writing. 

Over four months, the participants engaged in a variety of activities. They 
explored various forms of naturalistic assessment such as performance assessment, 
portfolios, retellings, holistic scoring, and miscue analysis. Windows into Literacy, 
(Rhodes & Shankin, 1 993) was the class textbook; additional readings were student 
selected from both books and journals on new literacy assessment methods as well as 
general readings on naturalistic assessment. 

Based on their particular teaching situations, the participants implemented 
various assessment procedures. For example, one English teacher implemented 
performance assessment in her classes; a third grade teacher tried holistic scoring with 
students' research reports. Weekly, we discussed issues of concern and influences on 
teacher confidence as they implemented these new assessment methods in their 
classrooms. These discussions were often sparked by a participant response to a reading 
or after attempts at implementation. I took scratch notes during these discussion 
(Sanjek, 1990). The notes were later developed more fully into fieldnotes as I recorded 
my impressions and reflected on the week's class. 

In addition to the fieldnotes, data were collected at the beginning of the class from 
a questionnaire which focused on participants' prior experiences with naturalistic 
assessment An end of class interview with each participant was recorded and 
transcribed. The interview focused on what the participants thought they would do with 
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naturalistic assessment in the future and how they viewed new assessntent procedures in 
terms of teacher empowerment, a strong theme which emerged from the class 
discussions and written explanations. 

The participants also completed confidence rating and wrote narratives 
explaining their ratings five times throughout the semester. Several researchers have 
suggested an interrelationship between teachers' self-efficacy or confidence and the 
success of an educational innovation (Guskey,1988; Ohihausen, Meyerson, & Sexton, 

1 992). The participants rated themselves on a 0 (least confident) to 9 (most confident) 
scale. They then wrote explanations for each rating. The narratives provided insights as 
to why a particular rating was given, what influenced the teachers' confidence, as well as 
what concerns the teachers had during the class. Table 1 presents the a summary of the 
data collected. 


Insert Table 1 about here 


Finally, the participants responded pre and post to the Stages of Concern 
Questionnaire (SoCQ) (Hord, Rutherford, Huling-Austin, & Hall, 1 987). This 
instrument identifies individual teachers' concerns about an educational innovation. The 
35-item instrument is written generically so that the respondents may focus on a 
specific innovation; in this case, the participants considered naturalistic assessment as 
the educational innovation. The concerns are classified as either "self" (how an 
innovation affects a teacher personally) "task" (how an innovation affects classroom 
management and implementation); and "impact" (how an innovation affects students, 
collaboration with colleagues, and refocusing to adapt the innovation). Pre and post 
measures allowed for a comparison of change over the four months. 

Data Analysis 

The scoring of the SoCQ according to the procedures outlined in Taking Charge of 
C/»nge(Hord, Rutherford, Huling-Austin, & Hall, 1987) resulted in pre and post 
profiles for each participant. A group profile was developed using mean scores to 
determine any overall pattern of concerns. 



7 


i 


Teachers' Concerns and Confidence 7. 

The constant comparative method (Glaser & Strauss, 1 967; Corbin and Strauss, 
1 990) was used to analyze narrative data from the confidence explanations, interviews, 
and class discussion fieldnotes. By repeatedly reading the data, ten factors were 
identified as categories which influenced the participants and futher organized into four 
generic categories: teacher uniqueness (teacher mood/nK)tivation, knowledge of 
procedures); professional development (conferences, university courses); significant 
others (students in their classes, administrators, colleagues); and teaching situation 
(opportunities to experiment, curricular demands, school district requirements). 
Teachir^ situation and significant others may be further refined into a core category: 
teaching environment. 


Group Results 

The results from the SoCQ, Figure 1 , indicated that at the beginning of the class, 
most of the participants were concerned about acquiring accurate information on 
naturalistic assessment and how its implementation would personally affect them in 
their classroom or school. By the end of the class, most of the participants were 
concerned about the impact of naturalistic assessment on their students; several were 
concerned about collaboration with colleagues in their schools. This general shift in 
concerns from "self" to "impact" is typical of the developmental nature of concerns 
(Hord, Rutherford, Huling-Austin, & Hall, 1987). 


Insert Figure 1 about here 


The confidence ratings showed that the participants, in general, were confident in 
their own abilities to implement naturalistic assessment at the beginning of the class 
(mean 6.2 ) and even more so at the end of the class (mean 8.9). The general high level 
of confidence at the beginning of the class may be attributed to their prior courses and 
experiences in the classroom. The interviews and written narratives further clarified 
their confidence ratings; they tended to rise after the teachers had been successful with 
a new method or procedure. Of note is the fact that there were individual dips in 
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confidence over the four month period; these dips are best understood when the 
participants are viewed as individuals 

A major theme emerged from all the data sources: empowerment. This theme was 
interrelated to the generic categories identified as influencing teacher confidence. If die 
teaching environment was supportive, with administrators or colleagues encouraging the 
teachers to experiment, the participants felt confident and empowered. When too many 
curricular demand and school district requirements prevented teachers from 
experimenting, coupled with non supportive colleagues and/or administrators, the 
participants felt restricted and less confident. Through class discussions and narratives, 
it was obvious that the participants were confident in their own abilities to use 
naturalistic assessment successfully in their classrooms. They felt more professional 
when they chose the procedures to use to determine student progress. Martha, a primary 
grade teacher stated: 


With alternative assessment, I don’t feel like someone is coming into 
my classroom and saying to me: ’Well, this is what you need to do and 
this is what has to be done’, instead, they are saying: ’You’re a pro* 
fessional and you know best for the students, therefore, you have the 
freedom to do what is needed’. 

The participants also saw naturalistic assessment as empowering students as well 
as themselves. They described the effect naturalistic assessment had on their students in 
terms of involvement and ownership. Frank, a middle school English teacher, echoed 
much of the group’s sentiments: 

By being able to use naturalistic assessment I feel much more 
comfortable in the classroom. I feel that I’m able to reach out help 
and kids more on an individual basis then I ever could before. I’m able 
to alleviate a lot of that tension because I’m basing their evaluation on 
what the kids are capable of doing and what they doing within those 
capabilities. The biggest asset of naturalistic assessment is that I 
don’t have to have a class criteria; I have an individual criteria and 
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I have individual standards. That's one part. Another part is it provides 
nie with a much more relaxed environment in the classroom. The kids 
aren’t afraid to come in the classroom. Research tells us that the 
environment plays a big part in how well the kids are going to learn. 

That’s another big plus of naturalistic assessment; it lets you provide 
a freer kind of environment. We don’t always have to answer the questions 
at the chapter. We can do things like a discussion. We can do things like a 
presentation. We can make collages and projects that represent learning. 

We can use different parts of our talents. 

We have to ask ourselves as teachers: What do we want, what are 
we looking for? Are we looking for an easy way to find out whether 
these kids may have comprehended something or are we interested in 
expanding the kids understanding to where they can show us symbolism. 

The participants were not confident, however, about the success of naturalistic 
assessment in their school and school district. Participants often asked during class 
discussions: "How can we convince parents, district level administrators, members of 
the school board, and the general public to let go of standardized tests? How can we use 
portfolios and not still have to use the traditional report cards?" The nine teachers felt 
somewhat powerless to influence district level administrators and school board members 
in a way that district wide change in assessment procedures would occur. 

Thomas, the principal, however, had a different perspective. At his building 
level, he had introduced naturalistic assessment procedures to his faculty and was 
working with his teachers as they progressed through the change process. He also was 
involved in an assessment task force which included parents, nfembers of the business 
community, and high level district decision-makers. Thomas felt he was in a position to 
influence decisions at the district level but was frustrated the resistance he faced 
primarily from the district decision-makers. Through class discussions, he intimated 
that there was some movement toward more openness to naturalistic assessment but 
there was little chance in the near future that naturalistic level would replace the 
established standardized testing policies already in place at the district level. 
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Individual Results - Two Portraits 

Individual portraits were developed for each participant. Two teachers are 
presented here as examples of the importance of a supportive teaching environment and 
its interrelationship with empowerment. 

Martha is a primary grade teacher who has taught for four years. Her teaching 
assignment during 1993-1994 was a combination first and second grade class. Martha 
had previously taken a course in whole language and had attended many inservices at her 
school. She stated that she works in a very supportive environment; her principal 
encouraged the use of portfolios which were transferred year to year with the students. 
At the beginning of the class, Martha's SoCQ revealed her main concern to be 
collaboration with colleagues (Fig. 2). This concern intensified at the end of the 
semester. 


Insert Figure 2 about here 


The interviews and narratives revealed that while naturalistic assessment was 
part of her school and that all teachers used portfolios, from Martha’s perspective, she 
and her colleagues had been "just on the surface level" and that there was much more to 
new assessment procedures than she thought at first. Martha's confidence ratings were 
7, 5, 6, 6, and 7. She attributed the dip in confidence from the first to last rating to her 
reexamination of her own classroom uses of portfolios as a result of reading the text and 
journal articles as well as her attempt to use rubrics to evaluate student writing. At the 
end of the class she stated: 

I still don't feel 100% confident but in a way I'm glad. I am very much 
aware of the fact that you can't have total confidence in something until 
you do It. Right now I can say I'm confident enough to give new assessment 
forms a try. 

Martha looked forward to refining the use portfolios in her classroom next year. 
She was eager to share her new knowledge of rubrics with colleagues at her school. As 
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stated previously, Martha saw naturalistic assessment as an empowering innovation. She 
had worked to gain knowledge, experimented in her own classroom, and was confident in 
her own abilities. By teaching at a school where both the principal and colleagues 
supported naturalistic assessment, Martha's confidence was validated and success became 
a given. 


Diane is a middle school reading and English teacher who taught for three years. 
She stated during a class discussion that while she taught in a middle school her 
principal and colleagues continued to embrace a junior high school perspective. They 
were traditional in the approaches to instruction and assessment and had made little 
progress toward an integrated curricular model used in many middle schools. Diane's 
principal allowed her to experiment but had no particular commitment to naturalistic 
assessment 

Diane's preservice education was holistic in nature; she felt that she didn't need 
to "change beliefs much, Just needed to know rrare about" naturalistic assessment. Diane 
used checklists as an assessment tool and was attempting, for the first time, to use 
student portfolios. Diane's pre SoCQ showed that she was, indeed, most concerned with 
"self" as shown by the awareness, information, and personal high scores (Fig. 3). 
Throughout the semester Diane focused on learning nx>re about naturalistic assessment 
through reading various articles, a textbook on evaluating writing,^ and keeping a double 
entry journal In which she reflected upon key points in her reading. As Figure 3 shows, 
the "self scores dropped on the post SoCQ as did the task (management) score. These 
changes may be attributed to both Diane's confidence with her new knowledge as well as 
her classroom success with portfolios. Two of the three scores related to impact rose on 
the post test Diane became increasingly concerned about the consequences of 
naturalistic assessment in terms of aligning instruction and assessment. She stated, "I 
would like to begin to use it for adapting my instruction to match students' needs and for 
students to see their own progress." Diane was also concerned about collaboration with 
her colleagues; she felt professionally responsible to "go out and teach the other teachers 
at school and hope they're willing to listen and willing to change." 


Insert Figure 3 about here 
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Diane's confidence ratings were 5, 6, 7, 8, and 9; they displayed a steady increase 
over the four months. Her written explanations focused on the personal concerns and 
validated the large "self" change from the beginning to the end of the class on the SoCQ. 
She explained her last confidence rating: 

I feel confident in my ability to use naturalistic assessment in my own 
classroom because I feel safe experimenting there. I know I still have 
a lot to learn but I feel that I can do it. Attitude affects confidence and I 
feel that as long as I am willing to find out more about naturalistic assess- 
ment that my confidence in using it will remain strong. 

At the post interview Diane told me that she would be going to a high school next 
year. She would be teaching with another participant in a school which openly supported 
naturalistic assessment. While Diane was anxious to work with older students she felt 
that her non-supportive colleagues and indifferent principal at the middle school 
influence greatly her decision to move to a new school. 

Discussion 

While no attempt is made to generalize to the population of teachers using 
naturalistic assessment, this study provided insights into how individual educators cope 
with educational change and innovations. The findings of this study suggest that in order 
for changes to occur in assessment on a large scale, individual teachers must be 
supported throughout the change process and their concerns acknowledge and addressed. 
Support for some teachers may mean the presence of just one other colleague who shares 
the same perspective; for others, support may need to be complete administrative 
endorsement and encouragement. The irxlividual portraits showed how teachers differ in 
their concerns for naturalistic assessment as an educational innovation. These 
differences must be addressed over time as teachers pass through the various stages. The 
teachers in this study felt personally empowered by naturalistic assessment but were 
not professionally at a point where they felt could greatly influence decision-makers. 

For these participants, district wide change may be sparked when more teachers take 
political action through their professional organizations. In addition, as more teacher 
education programs incorporate naturalistic assessment procedures into methods and 
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measurement courses, future teachers will enter the classroom empowered with the 
knowledge of naturalistic assessment. Taken together, naturalistic assessment and 
teacher empowerment may well become the cornerstone for restructuring our schools. 
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Table 1 

Summary Data for Participants 


Name 

Level 

Yrs. Exper. 

1st 

Confidence Ratings 
2nd 3rd 4th 

5th 

Diane 

middle 

03 

5 

6 

7 

8 

9 

Ada 

elementary 

12 

8 

7 

7 

7 

8 

Rebecca 

middle 

06.5 

4 

4 

7 

7 

8 

Frank 

middle 

04 

6 

6 

6 

8 

9 

Thomas 

elementary 

10 

3 

3 

4 

6 

7 

Martha 

elementary 

04 

7 

5 

6 

6 

7 

Kate 

secondary 

11 

7 

7 

7 

7 

7 

Kelly 

elementary 

10.5 

7 

7 

7 

7 

8 

Helen 

secondary 

04 

9 

7 

7 

9 

9 

Toni 

secondary 

05 

6 

6 

7 

8 

8 
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PRE 

POST 


Awareness Informational Personal 

55.60 75.00 73.00 

48.70 58.40 62.80 


Management 

50.90 

53.40 


Consequence 

65.80 

69.60 


Collaboration 

78.30 

82.00 



Figure 1. Group pre and post Stages of Concern profile 



Refocusing 

55.10 

56.10 


lijpRE 

Spost 
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Awareness 
PRE 10 

POST 37 


Informational Personal 
5 28 

12 39 


Management Consequence Collaboration Refocusing 
18 24 84 34 

39 33 97 38 



Spre 

[Upost 


! 


I 

j 

I 


Figure 2. Martha's pre and post Stages of Concern profile 
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Awareness Informational Personal 
PRE 77 75 85 

POST 37 45 55 


Management Consequence Collaboration Refocusing 
69 38 64 60 

27 71 84 60 





[sIpre 

HI POST 


Figure 3. Diane's pre and post Stages of Concern profile 


